Multimodal automatic assessment of acute pain through facial videos and heart rate signals utilizing transformer-based architectures.

Accurate and objective pain evaluation is crucial in developing effective pain management protocols, aiming to alleviate distress and prevent patients from experiencing decreased functionality. A multimodal automatic assessment framework for acute pain utilizing video and heart rate signals is introduced in this study. The proposed framework comprises four pivotal modules: the Spatial Module, responsible for extracting embeddings from videos; the Heart Rate Encoder, tasked with mapping heart rate signals into a higher dimensional space; the AugmNet, designed to create learning-based augmentations in the latent space; and the Temporal Module, which utilizes the extracted video and heart rate embeddings for the final assessment. The Spatial-Module undergoes pre-training on a two-stage strategy: first, with a face recognition objective learning universal facial features, and second, with an emotion recognition objective in a multitask learning approach, enabling the extraction of high-quality embeddings for the automatic pain assessment. Experiments with the facial videos and heart rate extracted from electrocardiograms of the BioVid database, along with a direct comparison to 29 studies, demonstrate state-of-the-art performances in unimodal and multimodal settings, maintaining high efficiency. Within the multimodal context, 82.74% and 39.77% accuracy were achieved for the binary and multi-level pain classification task, respectively, utilizing 9.62 million parameters for the entire framework.

A Full Transformer-based Framework for Automatic Pain Estimation using Videos.

The automatic estimation of pain is essential in designing an optimal pain management system offering reliable assessment and reducing the suffering of patients. In this study, we present a novel full transformer-based framework consisting of a Transformer in Transformer (TNT) model and a Transformer leveraging cross-attention and self-attention blocks. Elaborating on videos from the BioVid database, we demonstrate state-of-the-art performances, showing the efficacy, efficiency, and generalization capability across all the primary pain estimation tasks.

Automatic assessment of pain based on deep learning methods: A systematic review.

BACKGROUND AND OBJECTIVE: The automatic assessment of pain is vital in designing optimal pain management interventions focused on reducing suffering and preventing the functional decline of patients. In recent years, there has been a surge in the adoption of deep learning algorithms by researchers attempting to encode the multidimensional nature of pain into meaningful features. This systematic review aims to discuss the models, the methods, and the types of data employed in establishing the foundation of a deep learning-based automatic pain assessment system. METHODS: The systematic review was conducted by identifying original studies searching digital libraries, namely Scopus, IEEE Xplore, and ACM Digital Library. Inclusion and exclusion criteria were applied to retrieve and select those of interest, published until December 2021. RESULTS: A total of one hundred and ten publications were identified and categorized by the number of information channels used (unimodal versus multimodal approaches) and whether the temporal dimension was also used. CONCLUSIONS: This review demonstrates the importance of multimodal approaches for automatic pain estimation, especially in clinical settings, and also reveals that significant improvements are observed when the temporal exploitation of modalities is included. It provides suggestions regarding better-performing deep architectures and learning methods. Also, it provides suggestions for adopting robust evaluation protocols and interpretation methods to provide objective and comprehensible results. Furthermore, the review presents the limitations of the available pain databases for optimally supporting deep learning model development, validation, and application as decision-support tools in real-life scenarios.